{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "06d8ee1f",
   "metadata": {},
   "source": [
    "# 第1章 嵌入模型与向量数据库"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35b40ebe",
   "metadata": {},
   "source": [
    "## 💡 学习目标\n",
    "\n",
    "1. 理解向量表征\n",
    "2. 向量与文本向量\n",
    "3. 向量间的相似度计算\n",
    "4. Embedding Models嵌入模型原理及选择\n",
    "5. 向量数据库概述及核心原理\n",
    "6. Chroma向量数据库核心操作\n",
    "7. Milvus向量数据库扩展学习"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6f09e65",
   "metadata": {},
   "source": [
    "## 1. 向量表征（Vector Representation）\n",
    "\n",
    "在人工智能领域，向量表征（Vector Representation）是核心概念之一。通过将文本、图像、声音、行为甚至复杂关系转化为高维向量（Embedding），AI系统能够以数学方式理解和处理现实世界中的复杂信息。这种表征方式为机器学习模型提供了统一的“语言”。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0955eafd",
   "metadata": {},
   "source": [
    "### 1.1. 向量表征的本质\n",
    "\n",
    "<p>向量表征的本质：万物皆可数学化</p>\n",
    "<p><b>（1）核心思想</b></p>\n",
    "\n",
    "- <b>降维抽象：</b>将复杂对象（如一段文字、一张图片）映射到低维稠密向量空间，保留关键语义或特征。<br>\n",
    "\n",
    "- <b>相似性度量：</b>向量空间中的距离（如余弦相似度）反映对象之间的语义关联（如“猫”和“狗”的向量距离小于“猫”和“汽车”）。\n",
    "\n",
    "<p><b>（2）数学意义</b></p>\n",
    "\n",
    "- <b>特征工程自动化：</b>传统机器学习依赖人工设计特征（如文本的TF-IDF），而向量表征通过深度学习自动提取高阶抽象特征。\n",
    "\n",
    "- <b>跨模态统一：</b>文本、图像、视频等不同模态数据可映射到同一向量空间，实现跨模态检索（如“用文字搜图片”）。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28a8f44b",
   "metadata": {},
   "source": [
    "### 1.2. 向量表征的典型应用场景\n",
    "\n",
    "<p><b>（1）自然语言处理（NLP）</b></p>\n",
    "\n",
    "- <b>词向量（Word2Vec、GloVe）：</b>单词映射为向量，解决“一词多义”问题（如“苹果”在“水果”和“公司”上下文中的不同向量）。\n",
    "\n",
    "- <b>句向量（BERT、Sentence-BERT）：</b>整句语义编码，用于文本相似度计算、聚类（如客服问答匹配）。\n",
    "\n",
    "- <b>知识图谱嵌入（TransE、RotatE）：</b>将实体和关系表示为向量，支持推理（如预测“巴黎-首都-法国”的三元组可信度）。\n",
    "\n",
    "<p><b>（2）计算机视觉（CV）</b></p>\n",
    "\n",
    "- <b>图像特征向量（CNN特征）：</b>ResNet、ViT等模型提取图像语义，用于以图搜图、图像分类。\n",
    "\n",
    "- <b>跨模态对齐（CLIP）：</b>将图像和文本映射到同一空间，实现“描述文字生成图片”或反向搜索。\n",
    "\n",
    "<p><b>（3）推荐系统</b></p>\n",
    "\n",
    "- <b>用户/物品向量：</b>用户行为序列（点击、购买）编码为用户向量，商品属性编码为物品向量，通过向量内积预测兴趣匹配度（如YouTube推荐算法）。\n",
    "\n",
    "<p><b>（4）复杂系统建模</b></p>\n",
    "\n",
    "- <b>图神经网络（GNN）：</b>社交网络中的用户、商品、交互事件均表示为向量，捕捉网络结构信息（如社区发现、欺诈检测）。\n",
    "\n",
    "- <b>时间序列向量化：</b>将股票价格、传感器数据编码为向量，预测未来趋势（如LSTM、Transformer编码）。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfa0737c",
   "metadata": {},
   "source": [
    "### 1.3. 向量表征的技术实现\n",
    "\n",
    "<p><b>（1）经典方法</b></p>\n",
    "\n",
    "- <b>无监督学习：</b>Word2Vec通过上下文预测（Skip-Gram）或矩阵分解（GloVe）生成词向量。\n",
    "\n",
    "- <b>有监督学习：</b>微调预训练模型（如BERT）适应具体任务，提取任务相关向量。\n",
    "\n",
    "<p><b>（2）前沿方向</b></p>\n",
    "\n",
    "- <b>对比学习（Contrastive Learning）：</b>通过构造正负样本对（如“同一图片的不同裁剪”为正样本），拉近正样本向量距离，推开负样本（SimCLR、MoCo）。\n",
    "\n",
    "- <b>多模态融合：</b>将文本、图像、语音等多模态信息融合为统一向量（如Google的MUM模型）。\n",
    "\n",
    "- <b>动态向量：</b>根据上下文动态调整向量（如Transformer的注意力机制），解决静态词向量无法适应多义性的问题"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "548831a7",
   "metadata": {},
   "source": [
    "## 2. 什么是向量\n",
    "\n",
    "向量是一种有大小和方向的数学对象。它可以表示为从一个点到另一个点的有向线段。例如，二维空间中的向量可以表示为 $(x,y)$，表示从原点 $(0,0)$ 到点 $(x,y)$ 的有向线段。\n",
    "<br />\n",
    "<img src=\"./assets/vector.png\" style=\"margin-left: 0px\" width=800px>\n",
    "<br />\n",
    "以此类推，我可以用一组坐标 $(x_0, x_1, \\ldots, x_{N-1})$ 表示一个 $N$ 维空间中的向量，$N$ 叫向量的维度。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59db2cf5",
   "metadata": {},
   "source": [
    "### 2.1 文本向量（Text Embeddings）\n",
    "\n",
    "1. 将文本转成一组 $N$ 维浮点数，即**文本向量**又叫 Embeddings\n",
    "2. 向量之间可以计算距离，距离远近对应**语义相似度**大小\n",
    "\n",
    "<br />\n",
    "<img src=\"./assets/embeddings.png\" style=\"margin-left: 0px\" width=800px>\n",
    "<br />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fe4f51b",
   "metadata": {},
   "source": [
    "### 2.2 文本向量是怎么得到的（选）\n",
    "\n",
    "1. 构建相关（正例）与不相关（负例）的句子对样本\n",
    "2. 训练双塔式模型，让正例间的距离小，负例间的距离大\n",
    "\n",
    "例如：\n",
    "\n",
    "<img src=\"./assets/sbert.png\" style=\"margin-left: 0px\" width=500px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7935670b",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "<b>扩展阅读：https://www.sbert.net</b>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd32525e",
   "metadata": {},
   "source": [
    "### 2.3 向量间的相似度计算\n",
    "\n",
    "<img src=\"./assets/sim.png\" style=\"margin-left: 0px\" width=500px>\n",
    "\n",
    "余弦相似度是通过计算两个向量夹角的余弦值来衡量相似性，等于两个向量的点积除以两个向量长度的乘积。\n",
    "\n",
    "<img src=\"./assets/cos.png\" style=\"margin-left: 0px\" width=800px>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "542abc9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install --upgrade numpy\n",
    "# !pip install --upgrade openai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "93f1da40",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from openai import OpenAI\n",
    "\n",
    "# 需要在系统环境变量中配置好相应的key\n",
    "# OpenAI key（1 代理方式 2 官网注册购买）\n",
    "# OPENAI_API_KEY\n",
    "\n",
    "# 阿里百炼配置\n",
    "# DASHSCOPE_API_KEY sk-xxx\n",
    "# DASHSCOPE_BASE_URL https://dashscope.aliyuncs.com/compatible-mode/v1\n",
    "client = OpenAI(\n",
    "    api_key=os.getenv(\"DASHSCOPE_API_KEY\"),  # 如果您没有配置环境变量，请在此处用您的API Key进行替换\n",
    "    base_url=\"https://dashscope.aliyuncs.com/compatible-mode/v1\"  # 百炼服务的base_url\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "418c2590",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from numpy import dot\n",
    "from numpy.linalg import norm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "853d5848",
   "metadata": {},
   "outputs": [],
   "source": [
    "def cos_sim(a, b):\n",
    "    '''余弦距离 -- 越大越相似'''\n",
    "    return dot(a, b)/(norm(a)*norm(b))\n",
    "\n",
    "def l2(a, b):\n",
    "    '''欧氏距离 -- 越小越相似'''\n",
    "    x = np.asarray(a)-np.asarray(b)\n",
    "    return norm(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ea3dc576",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_embeddings(texts, model=\"text-embedding-v1\", dimensions=None):\n",
    "    '''封装 OpenAI 的 Embedding 模型接口'''\n",
    "    if model == \"text-embedding-v1\":\n",
    "        dimensions = None\n",
    "    if dimensions:\n",
    "        data = client.embeddings.create(\n",
    "            input=texts, model=model, dimensions=dimensions).data\n",
    "    else:\n",
    "        data = client.embeddings.create(input=texts, model=model).data\n",
    "    return [x.embedding for x in data]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a057e03c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total dimension: 1536\n",
      "First 10 elements: [1.8666456937789917, -3.2402632236480713, 1.2597650289535522, -2.3682942390441895, 2.38277530670166, -3.507119655609131, 1.461685299873352, -4.431000709533691, -2.722412586212158, 1.1500697135925293]\n"
     ]
    }
   ],
   "source": [
    "test_query = [\"聚客AI - 用科技力量，构建智能未来！\"]\n",
    "vec = get_embeddings(test_query)[0]\n",
    "print(f\"Total dimension: {len(vec)}\")\n",
    "print(f\"First 10 elements: {vec[:10]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "9078e328",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Query与自己的余弦距离: 1.00\n",
      "Query与Documents的余弦距离:\n",
      "0.3391104764873069\n",
      "0.2688252041673728\n",
      "0.07006357800727327\n",
      "0.10452984187535314\n",
      "0.12916023809848173\n",
      "\n",
      "Query与自己的欧氏距离: 0.00\n",
      "Query与Documents的欧氏距离:\n",
      "109.51274483356156\n",
      "109.95511077641505\n",
      "128.00071081019382\n",
      "123.82186767906518\n",
      "134.32520931177118\n"
     ]
    }
   ],
   "source": [
    "query = \"国际争端\"\n",
    "\n",
    "# 且能支持跨语言\n",
    "# query = \"global conflicts\"\n",
    "\n",
    "documents = [\n",
    "    \"联合国就苏丹达尔富尔地区大规模暴力事件发出警告\",\n",
    "    \"土耳其、芬兰、瑞场典与北约代表将继续就瑞典“入约”问题进行谈判\",\n",
    "    \"日本岐阜市陆上自卫队射击内发生枪击事件 3人受伤\",\n",
    "    \"国家游泳中心（水立方）：恢复游泳、嬉水乐园等水上项目运营\",\n",
    "    \"我国首次在空间站开展舱外辐射生物学暴露实验\",\n",
    "]\n",
    "\n",
    "query_vec = get_embeddings([query])[0]\n",
    "doc_vecs = get_embeddings(documents)\n",
    "\n",
    "print(\"Query与自己的余弦距离: {:.2f}\".format(cos_sim(query_vec, query_vec)))\n",
    "print(\"Query与Documents的余弦距离:\")\n",
    "for vec in doc_vecs:\n",
    "    print(cos_sim(query_vec, vec))\n",
    "\n",
    "print()\n",
    "\n",
    "print(\"Query与自己的欧氏距离: {:.2f}\".format(l2(query_vec, query_vec)))\n",
    "print(\"Query与Documents的欧氏距离:\")\n",
    "for vec in doc_vecs:\n",
    "    print(l2(query_vec, vec))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b451b687",
   "metadata": {},
   "source": [
    "## 3. Embedding Models 嵌入模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "097b2eff",
   "metadata": {},
   "source": [
    "### 3.1. 什么是嵌入（Embedding）？\n",
    "\n",
    "嵌入（Embedding）是指非结构化数据转换为向量的过程，通过神经网络模型或相关大模型，将真实世界的离散数据投影到高维数据空间上，根据数据在空间中的不同距离，反映数据在物理世界的相似度。 "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de30cae8",
   "metadata": {},
   "source": [
    "### 3.2. 嵌入模型概念及原理\n",
    "\n",
    "<b>1. 嵌入模型的本质</b>\n",
    "\n",
    "嵌入模型（Embedding Model）是一种将离散数据（如文本、图像）映射到连续向量空间的技术。通过高维向量表示（如 768 维或 3072 维），模型可捕捉数据的语义信息，使得语义相似的文本在向量空间\n",
    "中距离更近。例如，“忘记密码”和“账号锁定”会被编码为相近的向量，从而支持语义检索而非仅关键词匹配。\n",
    "\n",
    "<b>2. 核心作用</b>\n",
    "\n",
    "- <b>语义编码：</b>将文本、图像等转换为向量，保留上下文信息（如 BERT 的 CLS Token 或均值池化。相似度计算：通过余弦相似度、欧氏距离等度量向量关联性，支撑检索增强生成（RAG）、推荐系统等应用。\n",
    "\n",
    "- <b>信息降维：</b>压缩复杂数据为低维稠密向量，提升存储与计算效率。\n",
    "\n",
    "<b>3. 关键技术原理</b>\n",
    "\n",
    "- <b>上下文依赖：</b>现代模型（如 BGE-M3）动态调整向量，捕捉多义词在不同语境中的含义。\n",
    "\n",
    "- <b>训练方法：</b>对比学习（如 Word2Vec 的 Skip-gram/CBOW）、预训练+微调（如 BERT）。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "385f808c",
   "metadata": {},
   "source": [
    "### 3.3. 主流嵌入模型分类与选型指南\n",
    "\n",
    "Embedding 模型将文本转换为数值向量，捕捉语义信息，使计算机能够理解和比较内容的\"意义\"。选择 Embedding 模型的考虑因素：\n",
    "\n",
    "| 因素          | 说明                         |\n",
    "| ------------ | -----------------------------|\n",
    "| **任务性质**  | 匹配任务需求(问答、搜索、聚类等) |\n",
    "| **领域特性**  | 通用vs专业领域(医学、法律等) \n",
    "| **多语言支持** | 需处理多语言内容时考虑 |\n",
    "| **维度**     | 权衡信息丰富度与计算成本 |\n",
    "| **许可条款**  | 开源vs专有服务 |\n",
    "| **最大Tokens** | 适合的上下文窗口大小 |\n",
    "\n",
    "最佳实践：为特定应用测试多个 Embedding 模型，评估在实际数据上的性能而非仅依赖通用基准。\n",
    "\n",
    "**1. 通用全能型**\n",
    "\n",
    "- <b>BGE-M3：</b>北京智源研究院开发，支持多语言、混合检索（稠密+稀疏向量），处理 8K 上下文，适合企业级知识库。\n",
    "- <b>NV-Embed-v2：</b>基于 Mistral-7B，检索精度高（MTEB 得分 62.65），但需较高计算资源。\n",
    "\n",
    "**2. 垂直领域特化型**\n",
    "\n",
    "- <b>中文场景：</b> BGE-large-zh-v1.5 （合同/政策文件）、 M3E-base （社交媒体分析）。\n",
    "- <b>多模态场景：</b> BGE-VL （图文跨模态检索），联合编码 OCR 文本与图像特征。\n",
    "\n",
    "**3. 轻量化部署型**\n",
    "\n",
    "- <b>nomic-embed-text：</b>768 维向量，推理速度比 OpenAI 快 3 倍，适合边缘设备。\n",
    "- <b>gte-qwen2-1.5b-instruct：</b>1.5B 参数，16GB 显存即可运行，适合初创团队原型验。\n",
    "\n",
    "**选型决策树：**\n",
    "\n",
    "1. 中文为主 → BGE 系列 > M3E；\n",
    "\n",
    "2. 多语言需求 → BGE-M3 > multilingual-e5；\n",
    "\n",
    "3. 预算有限 → 开源模型（如 Nomic Embed）\n",
    "\n",
    "**Embddding Leaderboard**\n",
    "\n",
    "<img src=\"./assets/Leaderboard.png\" width=\"1400px\">\n",
    "\n",
    "https://huggingface.co/spaces/mteb/leaderboard"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f64fa2d",
   "metadata": {},
   "source": [
    "### 3.4. OpenAI 新发布的两个 Embedding 模型\n",
    "\n",
    "2024 年 1 月 25 日，OpenAI 新发布了两个 Embedding 模型\n",
    "\n",
    "- text-embedding-3-large\n",
    "- text-embedding-3-small\n",
    "\n",
    "其最大特点是，支持自定义的缩短向量维度，从而在几乎不影响最终效果的情况下降低向量检索与相似度计算的复杂度。\n",
    "\n",
    "通俗的说：**越大越准、越小越快。** 官方公布的评测结果:\n",
    "\n",
    "<img src=\"./assets/mteb.png\" style=\"margin-left: 0px\" width=800px>\n",
    "\n",
    "注：[MTEB](https://huggingface.co/blog/mteb) 是一个大规模多任务的 Embedding 模型公开评测集"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08e75b9d",
   "metadata": {},
   "source": [
    "### 3.5. 嵌入模型使用\n",
    "\n",
    "1. 使用 API 调用方式"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "6ee99980",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"data\":[{\"embedding\":[-0.061863791197538376,-0.036015111953020096,0.009983918629586697,-0.057213194668293,-0.02076546847820282,0.01999487541615963,0.03874598443508148,0.10085312277078629,-0.008969979360699654,0.14330336451530457,0.06408093869686127,0.006019417196512222,0.004180808085948229,-0.03574472665786743,-0.04888537526130676,-0.047722723335027695,-0.012349775992333889,-0.019724491983652115,-0.015655217692255974,-0.016520444303750992,0.011457509361207485,0.011484548449516296,0.0392056368291378,0.03498765081167221,-0.03523099794983864,-0.03977344557642937,0.01606079190969467,0.0004452881694305688,0.019021494314074516,-0.02175236865878105,0.0233070757240057,0.018737591803073883,0.025632375851273537,0.00515080988407135,0.01829145848751068,0.03239196911454201,-0.0014820407377555966,-0.020008394494652748,-0.018724072724580765,-0.04139574617147446,0.018426649272441864,-0.00003699292938108556,0.018183303996920586,-0.025875721126794815,-0.017399191856384277,-0.009767611511051655,0.020251739770174026,0.021495504304766655,-0.05112956091761589,-0.04983171820640564,0.03306792676448822,-0.04107128456234932,-0.0019062052015215158,0.010092072188854218,-0.039097484201192856,0.023482825607061386,-0.005454991478472948,0.031959354877471924,0.028714749962091446,0.054725661873817444,-0.008057434111833572,0.014465528540313244,0.028958095237612724,0.05499604716897011,-0.047857917845249176,0.003775232471525669,0.041801322251558304,0.035420265048742294,-0.024010073393583298,-0.046560075134038925,-0.010862665250897408,-0.03355461731553078,0.06467578560113907,0.02345578745007515,-0.00834809709340334,0.03552842140197754,0.001941693015396595,-0.00701645715162158,-0.05072398483753204,-0.003195597557350993,0.00880774948745966,-0.01084914617240429,0.0004123351536691189,-0.016114868223667145,0.0028542380314320326,0.010443571023643017,-0.00788168516010046,-0.005404294468462467,0.008550885133445263,0.0730036050081253,-0.05072398483753204,0.004261923022568226,-0.012261901050806046,0.016439329832792282,-0.02413174696266651,0.0019095849711447954,-0.03698849305510521,-0.004934502765536308,0.05269778519868851,0.019589299336075783,-0.04012494161725044,-0.0369073748588562,-0.044099584221839905,0.0543200857937336,0.012126709334552288,0.014560163021087646,0.00945666991174221,-0.07938465476036072,-0.019102608785033226,0.04531630873680115,-0.05199478566646576,-0.010997856967151165,0.004711436107754707,0.047019727528095245,0.021333275362849236,-0.020359894260764122,-0.023739689961075783,0.0019957697950303555,-0.01484406553208828,0.0194541085511446,-0.008294020779430866,0.019116127863526344,-0.0020802647341042757,-0.012065873481333256,0.027322273701429367,0.011281760409474373,0.00469115749001503,0.00006484984623966739,0.021481985226273537,-0.03652884066104889,-0.03682626038789749,-0.029715169221162796,0.026970775797963142,-0.010747752152383327,0.0011660297168418765,-0.023401709273457527,0.00034347179462201893,0.023117806762456894,-0.04620857536792755,0.031797125935554504,-0.017507344484329224,0.021549582481384277,0.054725661873817444,-0.005120391491800547,-0.046181537210941315,-0.014938700012862682,-0.0024571120738983154,-0.022428328171372414,-0.005225165281444788,0.01791292056441307,-0.049939870834350586,-0.0018538183066993952,-0.010369215160608292,0.028579557314515114,0.033338312059640884,-0.046343766152858734,0.027105966582894325,0.03263531252741814,-0.021481985226273537,0.03598807007074356,0.02798471413552761,0.052941132336854935,-0.01722344197332859,0.03469023108482361,0.019494665786623955,-0.018021075055003166,-0.018724072724580765,-0.002506118966266513,0.004062515217810869,-0.001890996121801436,-0.03574472665786743,-0.01426274050027132,0.013215003535151482,-0.056564271450042725,0.02107641100883484,0.00949046853929758,0.0666225478053093,0.03912452235817909,0.02276630885899067,-0.09106523543596268,-0.00041529248119331896,0.02214442566037178,0.09604029357433319,-0.01392476074397564,0.06283717602491379,-0.009199805557727814,0.03766445070505142,0.021400870755314827,0.04477554187178612,0.03098597377538681,0.00038149452302604914,0.02890401892364025,0.01211319025605917,0.044721465557813644,0.023482825607061386,-0.02491585910320282,0.00811827089637518,0.059862952679395676,0.10052866488695145,0.013309638015925884,0.014979257248342037,-0.009706775657832623,0.005911263637244701,0.0017203163588419557,-0.015939120203256607,-0.006752833258360624,0.057862114161252975,0.013870684430003166,-0.0020312576089054346,-0.014925180934369564,-0.007509907707571983,-0.01944058947265148,-0.0443158894777298,-0.01700713485479355,-0.04096313193440437,0.01118712592869997,-0.009321478195488453,0.01357326190918684,-0.059159956872463226,0.04482961818575859,-0.05167032778263092,0.015006296336650848,-0.03206750750541687,0.015398352406919003,-0.015249641612172127,-0.03309496492147446,-0.02222554013133049,0.022712232545018196,0.03122931905090809,-0.04112536087632179,-0.020400451496243477,0.00023743069323245436,-0.030958935618400574,0.008023636415600777,-0.0015792098129168153,0.007854647003114223,-0.018480727449059486,0.023023173213005066,-0.04583003744482994,-0.02099529467523098,0.0030485764145851135,0.01761549897491932,-0.0475064180791378,-0.007523426786065102,-0.009740573354065418,0.008219664916396141,-0.0606200285255909,-0.025591818615794182,-0.01961633749306202,0.00021250470308586955,-0.08035803586244583,0.012261901050806046,0.026010913774371147,0.04734418913722038,-0.05199478566646576,0.008740153163671494,-0.02007598988711834,-0.007746493443846703,0.012525524944067001,-0.009314718656241894,-0.030418166890740395,0.0027308755088597536,0.0027900219429284334,-0.011261481791734695,0.02499697357416153,0.026416487991809845,0.05523939058184624,0.07657266408205032,-0.01392476074397564,-0.023793766275048256,-0.0027511543594300747,0.017858844250440598,0.02760617621243,-0.012430891394615173,-0.030580397695302963,-0.005187987349927425,0.003967880737036467,-0.07819496840238571,0.03552842140197754,-0.04628968983888626,0.03698849305510521,0.005313039757311344,-0.022860942408442497,0.01815626583993435,0.012376814149320126,0.025740528479218483,0.01869703270494938,0.03452799841761589,0.01631765626370907,0.023266518488526344,0.01623654179275036,0.04745234176516533,0.022401290014386177,0.00371101638302207,-0.02622721903026104,0.034582074731588364,0.0640268623828888,0.0435047373175621,-0.018034594133496284,-0.028633635491132736,-0.009118691086769104,0.027430428192019463,-0.012775630690157413,-0.018751110881567,-0.020332854241132736,0.011065453290939331,-0.00333585892803967,-0.07208430022001266,0.06056595221161842,-0.020008394494652748,-0.04299100860953331,-0.027322273701429367,0.0018825465813279152,0.042558394372463226,-0.00014269078383222222,-0.015141488052904606,-0.031499702483415604,-0.017939958721399307,0.008726634085178375,0.005066314712166786,-0.04788495600223541,-0.029931476339697838,-0.022942058742046356,-0.002512878505513072,0.09355276077985764,0.02914736419916153,-0.015479467809200287,-0.015222602523863316,-0.026037951931357384,-0.018737591803073883,-0.030850781127810478,0.039557136595249176,0.015560582280158997,0.012863504700362682,0.0006662423838861287,0.015290198847651482,-0.010004197247326374,-0.008321058936417103,0.01042329240590334,0.004738474730402231,-0.00934175681322813,0.009585102088749409,-0.0013451589038595557,-0.0184536874294281,-0.006384435575455427,0.021968675777316093,-0.025118647143244743,0.042179860174655914,0.026200180873274803,0.006566944532096386,-0.005830148700624704,0.02968813106417656,-0.05099436640739441,-0.001847058767452836,0.008408932946622372,-0.03669106960296631,0.018642956390976906,0.0038800062611699104,0.027484504505991936,0.015398352406919003,-0.008307539857923985,0.02960701659321785,-0.016709713265299797,-0.03650180250406265,0.0009108551312237978,-0.06716331094503403,-0.000472749030450359,-0.021657735109329224,-0.05705096200108528,0.05672650411725044,-0.028498442843556404,-0.07067830115556717,0.014735911972820759,-0.004569484852254391,-0.015601140446960926,-0.024483244866132736,0.03190527856349945,0.002600753214210272,-0.002288122195750475,-0.02092769928276539,0.029715169221162796,0.015952639281749725,-0.01360706053674221,-0.0221579447388649,0.04099017009139061,-0.035176921635866165,0.014100510627031326,0.028282135725021362,0.015033334493637085,-0.0057794516906142235,-0.02191459946334362,-0.002791711827740073,-0.03052632138133049,-0.0054752700962126255,0.05126475170254707,-0.020103028044104576,0.034717269241809845,0.030553359538316727,-0.0011542004067450762,0.054482318460941315,0.024699551984667778,0.01276211068034172,0.015858003869652748,-0.015939120203256607,-0.018129227682948112,0.005018997471779585,0.004353177733719349,-0.004701296798884869,-0.027268197387456894,0.017507344484329224,-0.06413501501083374,-0.009909562766551971,0.05058879405260086,0.04407254606485367,-0.008719874545931816,0.011977998539805412,0.016574520617723465,-0.019102608785033226,0.021427908912301064,0.031121164560317993,-0.04445108026266098,-0.026497604325413704,-0.04783087968826294,0.03209454566240311,-0.05569904297590256,0.14676427841186523,-0.027795445173978806,-0.018359053879976273,-0.0072124856524169445,0.01899445615708828,-0.09479653090238571,-0.07408513873815536,0.03912452235817909,0.04604634642601013,-0.018778149038553238,-0.005258962977677584,0.02553774230182171,0.012315978296101093,-0.040341250598430634,-0.017574941739439964,0.008909143507480621,0.0018740970408543944,0.02906624972820282,0.011606221087276936,-0.03685329854488373,0.058781418949365616,-0.02968813106417656,0.01944058947265148,0.00957158301025629,-0.008834787644445896,-0.016412291675806046,-0.05129178985953331,0.0026649695355445147,-0.0010975889163091779,0.0100785531103611,-0.0005602012388408184,-0.013181205838918686,0.038556717336177826,0.006637920159846544,0.03368981182575226,0.018656475469470024,-0.013478628359735012,-0.01768309436738491,0.0009869005298241973,-0.01369493454694748,0.005008858162909746,0.01303249504417181,0.026037951931357384,-0.017196403816342354,-0.05759172886610031,0.032878659665584564,-0.018629437312483788,0.004160529468208551,-0.029634054750204086,0.012640438042581081,-0.001689053256995976,-0.046100422739982605,0.022009234875440598,-0.0183049775660038,-0.0030840642284601927,0.005816629622131586,0.006830568425357342,-0.00040599802741780877,0.015114448964595795,-0.04439700394868851,0.010342177003622055,0.01164677832275629,-0.0013240352272987366,0.010220504365861416,-0.008246703073382378,0.04966948926448822,-0.029039209708571434,-0.0015471017686650157,-0.005982239730656147,-0.016561001539230347,-0.001241230173036456,-0.04144982248544693,0.01453312486410141,0.02944478578865528,0.05099436640739441,-0.0538063570857048,-0.0036873577628284693,-0.019102608785033226,-0.021346794441342354,-0.07808681577444077,0.006195166613906622,0.046262651681900024,-0.011552143841981888,0.015492986887693405,-0.04807422310113907,-0.021657735109329224,0.022955577820539474,0.004765512887388468,0.015222602523863316,0.015939120203256607,-0.009524266235530376,0.02882290445268154,-0.006641299929469824,-0.04866906628012657,-0.06894784420728683,0.012072633020579815,-0.016871944069862366,-0.02413174696266651,-0.012484967708587646,0.003320649964734912,0.03222973644733429,-0.005860567092895508,-0.001089984318241477,0.09436391294002533,0.012059113942086697,0.0036299012135714293,0.009334997273981571,0.016479887068271637,0.011619740165770054,-0.025199761614203453,0.07608597725629807,0.008550885133445263,-0.0017059522215276957,-0.024510283023118973,0.056780580431222916,0.05331966653466225,0.06051187589764595,0.042179860174655914,0.036177340894937515,-0.0509132519364357,-0.005434712395071983,0.033040888607501984,0.028390290215611458,-0.00373805477283895,0.001682293601334095,0.01037597469985485,0.003660319373011589,0.02175236865878105,-0.023482825607061386,0.041557975113391876,-0.018710553646087646,-0.02667335234582424,-0.0013840266037732363,0.03823225572705269,-0.015587621368467808,0.025118647143244743,-0.005681437905877829,-0.013769290409982204,0.01337723433971405,-0.016358215361833572,-0.056239813566207886,-0.015371314249932766,-0.004606662318110466,0.007956041023135185,-0.028714749962091446,-0.05415785685181618,-0.004653979558497667,-0.03222973644733429,0.04466738924384117,0.02069787308573723,0.0019180343952029943,-0.006333738565444946,-0.006475689820945263,0.05058879405260086,-0.008861825801432133,-0.03742110729217529,-0.03331127390265465,-0.0429098941385746,-0.0005209111259318888,0.011781970039010048,-0.0395030602812767,-0.014249221421778202,-0.027416909113526344,-0.006648059468716383,-0.022022753953933716,0.050102099776268005,-0.019102608785033226,-0.05296817049384117,-0.022401290014386177,-0.01992728002369404,-0.0035183681175112724,0.04353177919983864,-0.04042236506938934,-0.023131325840950012,0.009869005531072617,-0.02192811854183674,-0.01484406553208828,0.019035013392567635,0.01130879856646061,0.0518055185675621,-0.0018436788814142346,-0.022590558975934982,-0.009632419794797897,0.014506085775792599,0.019548742100596428,0.029498862102627754,-0.03736702725291252,0.007063774392008781,0.011227683164179325,-0.003138141008093953,-0.021549582481384277,-0.005826768931001425,0.02721412107348442,-0.0283091738820076,-0.027592657133936882,0.04804718494415283,-0.014749431051313877,-0.04374808445572853,-0.0466141514480114,-0.009990678168833256,0.03166193142533302,-0.0015580861363559961,-0.0002334171877009794,0.005728754680603743,0.034257616847753525,0.04853387549519539,0.006935342215001583,-0.03298681229352951,-0.01631765626370907,0.00520826643332839,0.01384364627301693,-0.0005344303208403289,0.048939451575279236,-0.027268197387456894,0.03228381648659706,0.04420773684978485,-0.004103072918951511,-0.033176083117723465,0.003585963975638151,-0.00799659825861454,0.041260555386543274,-0.05507716163992882,-0.04069274663925171,0.04190947487950325,0.04082794114947319,0.037934836000204086,0.008199386298656464,0.05167032778263092,0.042639512568712234,-0.018588880077004433,0.031797125935554504,0.0010071792639791965,-0.057483576238155365,0.004177428316324949,-0.036177340894937515,-0.058024343103170395,-0.05672650411725044,0.006137710064649582,0.04153093695640564,0.034798383712768555,-0.008294020779430866,-0.033419426530599594,0.008990257978439331,0.01072747353464365,-0.046100422739982605,-0.019494665786623955,-0.001299531664699316,-0.03623141720890999,-0.003226015716791153,0.05226517096161842,0.03212158381938934,-0.01577688939869404,0.021427908912301064,0.03744814544916153,-0.0455055795609951,-0.0283091738820076,0.004424153361469507,-0.05764580890536308,0.023726170882582664,0.006793390493839979,-0.010930261574685574,-0.038718946278095245,-0.0066514392383396626,-0.010937021113932133,0.04050347954034805,-0.0091524887830019,0.017318077385425568,0.05672650411725044,-0.007205726113170385,0.017980515956878662,-0.0015631557907909155,-0.06478393822908401,0.010348936542868614,0.04923687502741814,0.01607431098818779,-0.007394994609057903,0.006137710064649582,-0.013390753418207169,-0.014141067862510681,0.046722304075956345,0.027173561975359917,-0.08690132200717926,-0.02192811854183674,0.008016876876354218,-0.05683465674519539,-0.014911661855876446,-0.00880774948745966,-0.010666637681424618,0.00945666991174221,-0.016506925225257874,0.006539905909448862,-0.006161368917673826,-0.013471868820488453,-0.009240363724529743,-0.009172767400741577,0.06159340962767601,0.020900661125779152,0.032878659665584564,0.016804346814751625,-0.04569484665989876,-0.012937860563397408,-0.0020261879544705153,-0.0070975725539028645,-0.0019146546255797148,0.024388611316680908,-0.014019395224750042,0.006181647535413504,0.01899445615708828,-0.016736751422286034,0.02444268763065338,-0.03531211242079735,-0.05029137060046196,-0.01814274676144123,0.028849942609667778,-0.014019395224750042,-0.07792458683252335,0.005684817675501108,0.044559236615896225,-0.008665798231959343,0.15628178417682648,0.018277939409017563,0.019102608785033226,0.02806582860648632,0.0077126952819526196,-0.05234628543257713,-0.023401709273457527,-0.02890401892364025,-0.11247962713241577,0.01944058947265148,-0.0006117431330494583,0.01569577492773533,0.033581655472517014,-0.021495504304766655,0.001458382117561996,-0.031634893268346786,0.04177428409457207,-0.0032581237610429525,-0.017331596463918686,-0.01310684997588396,0.018669994547963142,0.04107128456234932,0.03158081695437431,0.06278309971094131,0.017899401485919952,0.029579978436231613,-0.023198923096060753,-0.06938046216964722,0.03628549352288246,0.010930261574685574,-0.018710553646087646,0.04523519426584244,-0.0183049775660038,0.014127548784017563,0.003704256843775511,0.010240782983601093,-0.007340917829424143,0.0001309671060880646,-0.08111511170864105,-0.007678897585719824,0.020265258848667145,0.012430891394615173,0.0016299068229272962,-0.03122931905090809,0.01003123540431261,0.09598621726036072,0.012512005865573883,0.008909143507480621,0.050399523228406906,-0.012376814149320126,0.06208010017871857,-0.05175144225358963,-0.02898513339459896,0.0718139111995697,-0.036096226423978806,0.009355276823043823,0.01783180609345436,0.008408932946622372,-0.022793347015976906,-0.0443158894777298,0.015087410807609558,-0.03036409057676792,-0.0002813680621329695,0.03731295093894005,0.022969096899032593,0.0233070757240057,-0.030337052419781685,-0.030039630830287933,-0.01745326817035675,-0.0033088207710534334,0.021941637620329857,-0.005225165281444788,-0.016750270500779152,-0.017710132524371147,-0.017520863562822342,-0.009213324636220932,-0.0006049835355952382,-0.004664118867367506,0.021481985226273537,0.027051890268921852,0.03544730320572853,-0.026119066402316093,0.04107128456234932,-0.0010832247789949179,-0.04807422310113907,0.04428885132074356,0.0132285226136446,0.021576620638370514,-0.03263531252741814,0.015790408477187157,-0.007111091632395983,-0.008185867220163345,-0.011957719922065735,-0.014370894059538841,0.004478230141103268,0.030634474009275436,-0.023171883076429367,0.01921076327562332,-0.02667335234582424,-0.006326979026198387,-0.014005876146256924,-0.034176502376794815,0.017791248857975006,0.028390290215611458,0.004383596125990152,-0.002791711827740073,0.0221579447388649,-0.01310684997588396,-0.05277889966964722,0.018940379843115807,-0.022942058742046356,-0.014289778657257557,-0.01107221283018589,0.005580043885856867,0.03852967917919159,-0.01280266884714365,0.018183303996920586,0.025091608986258507,-0.009334997273981571,-0.005478649865835905,-0.021319756284356117,0.04245024174451828,0.02621369995176792,-0.00336627708747983,0.04007086530327797,-0.0032953014597296715,0.005647639743983746,0.016723232343792915,-0.02345578745007515,0.026037951931357384,-0.01268099620938301,0.05445528030395508,-0.007780291140079498,-0.01030837930738926,-0.00017522131383884698,0.006333738565444946,0.004985199775546789,-0.022049792110919952,0.0014161346480250359,-0.010511166416108608,0.03812410309910774,0.028714749962091446,-0.0033966952469199896,-0.01003799494355917,0.02091418020427227,0.013512426055967808,-0.018332015722990036,-0.02599739283323288,0.02499697357416153,0.028119906783103943,0.012741832062602043,-0.0007946746191009879,0.010004197247326374,0.03309496492147446,-0.01576337032020092,0.00770593574270606,0.0017245410708710551,-0.013539464212954044,0.01003123540431261,0.040800902992486954,-0.029093287885189056,-0.04791199415922165,-0.0660277009010315,-0.0011212474200874567,-0.02822805941104889,-0.0706242248415947,-0.03301385045051575,0.01745326817035675,0.040097903460264206,0.006833948194980621,-0.004684397950768471,-0.009328237734735012,0.019805606454610825,0.028119906783103943,0.0068407077342271805,-0.022171463817358017,-0.007820848375558853,-0.03590695559978485,0.008368375711143017,0.015574101358652115,0.03298681229352951,0.0027663633227348328,-0.010991097427904606,0.005380635615438223,0.02069787308573723,0.04253135621547699,0.014343855902552605,0.02046804688870907,0.004508648533374071,-0.03228381648659706,-0.023888401687145233,-0.018507765606045723,-0.038178179413080215,-0.016344694420695305,-0.028714749962091446,-0.03752925992012024,-0.0058943647891283035,-0.012647198513150215,-0.01441145222634077,0.004670878406614065,-0.005711855832487345,0.003890145570039749,-0.02391543984413147,0.012552564032375813,-0.05423897132277489,0.005421193316578865,-0.04553261771798134,0.02246888540685177,0.0164528489112854,-0.011383154429495335,-0.01130203902721405,-0.007029976695775986,0.02269871160387993,0.006823808886110783,-0.013708454556763172,-0.02076546847820282,0.05307632312178612,-0.038800060749053955,-0.01569577492773533,-0.0592140331864357,0.04866906628012657,0.032175660133361816,0.007138129789382219,-0.013471868820488453,0.0532115139067173,0.04299100860953331,0.001609627972356975,0.01280942838639021,0.006043076049536467,0.00776677206158638,-0.05607758089900017,-0.030201859772205353,-0.004870286677032709,-0.021887561306357384,-0.10517926514148712,-0.01846720650792122,-0.017291037365794182,-0.01630413718521595,0.03185120224952698,-0.007442311849445105,0.00016212460468523204,-0.008158828131854534,0.023036692291498184,0.06013333797454834,-0.009713535197079182,0.010828867554664612,-0.06932638585567474,0.027416909113526344,0.04539742320775986,-0.006715655326843262,-0.0062086861580610275,-0.017277518287301064,-0.003673838684335351,-0.013640858232975006,0.034311693161726,0.008415693417191505,0.03953009843826294,0.0006599052576348186,0.0030080187134444714,-0.003371346974745393,0.017818287014961243,0.010700435377657413,-0.026511123403906822,0.008321058936417103,-0.005820009391754866,-0.02352338284254074,-0.0031280014663934708,-0.02460491843521595,0.018656475469470024,-0.018710553646087646,0.06391870975494385,-0.012498486787080765,-0.04088201746344566,0.02768729254603386,-0.004099693149328232,-0.013762530870735645,-0.03842152655124664,0.03912452235817909,-0.0014490877510979772,-0.02615962363779545],\"index\":0,\"object\":\"embedding\"}],\"model\":\"text-embedding-v4\",\"object\":\"list\",\"usage\":{\"prompt_tokens\":5,\"total_tokens\":5},\"id\":\"872c35dd-0947-9688-a261-3823ae36b7e4\"}\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "from openai import OpenAI\n",
    "\n",
    "client = OpenAI(\n",
    "    api_key=os.getenv(\"DASHSCOPE_API_KEY\"),  # 如果您没有配置环境变量，请在此处用您的API Key进行替换\n",
    "    base_url=\"https://dashscope.aliyuncs.com/compatible-mode/v1\"  # 百炼服务的base_url\n",
    ")\n",
    "\n",
    "completion = client.embeddings.create(\n",
    "    model=\"text-embedding-v4\",\n",
    "    input='聚客AI学院',\n",
    "    dimensions=1024,\n",
    "    encoding_format=\"float\"\n",
    ")\n",
    "\n",
    "print(completion.model_dump_json())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "135de10c",
   "metadata": {},
   "source": [
    "## 4. 向量数据库"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "139f065e",
   "metadata": {},
   "source": [
    "### 4.1. 什么是向量数据库？\n",
    "\n",
    "**向量数据库，是专门为向量检索设计的中间件！**\n",
    "\n",
    "**高效存储、快速检索和管理高纬度向量数据的系统称为向量数据库**\n",
    "\n",
    "一种专门用于存储和检索高维向量数据的数据库。它将数据（如文本、图像、音频等）通过嵌入模型转换为向量形式，并通过高效的索引和搜索算法实现快速检索。\n",
    "\n",
    "向量数据库的核心作用是实现相似性搜索，即通过计算向量之间的距离（如欧几里得距离、余弦相似度等）来找到与目标向量最相似的其他向量。它特别适合处理非结构化数据，支持语义搜索、内容推荐等场景。\n",
    "\n",
    "**核心功能：**\n",
    "\n",
    "- 向量存储\n",
    "\n",
    "- 相似性度量\n",
    "\n",
    "- 相似性搜索\n",
    "\n",
    "<img src=\"./assets/embedding.png\" width=\"1200px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c2013f0",
   "metadata": {},
   "source": [
    "### 4.2. 如何存储和检索嵌入向量？\n",
    "\n",
    "- 存储：向量数据库将嵌入向量存储为高维空间中的点，并为每个向量分配唯一标识符（ID），同时支持存储元数据。\n",
    "\n",
    "- 检索：通过近似最近邻（ANN）算法（如PQ等）对向量进行索引和快速搜索。比如，FAISS和Milvus等数据库通过高效的索引结构加速检索。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc7c873a",
   "metadata": {},
   "source": [
    "### 4.3 向量数据库与传统数据库对比\n",
    "\n",
    "1. 数据类型\n",
    "\n",
    "- 传统数据库：存储结构化数据（如表格、行、列）。\n",
    "- 向量数据库：存储高维向量数据，适合非结构化数据。\n",
    "\n",
    "2. 查询方式\n",
    "\n",
    "- 传统数据库：依赖精确匹配（如=、<、>）。\n",
    "- 向量数据库：基于相似度或距离度量（如欧几里得距离、余弦相似度）。\n",
    "\n",
    "3. 应用场景\n",
    "\n",
    "- 传统数据库：适合事务记录和结构化信息管理。\n",
    "- 向量数据库：适合语义搜索、内容推荐等需要相似性计算的场景。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec969afc",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "<b>澄清几个关键概念：</b><ul>\n",
    "    <li>向量数据库的意义是快速的检索；</li>\n",
    "    <li>向量数据库本身不生成向量，向量是由 Embedding 模型产生的；</li>\n",
    "    <li>向量数据库与传统的关系型数据库是互补的，不是替代关系，在实际应用中根据实际需求经常同时使用。</li>\n",
    "</ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea1c6555",
   "metadata": {},
   "source": [
    "### 4.4. 主流向量数据库功能对比\n",
    "\n",
    "<img src=\"./assets/vectordb.png\" style=\"margin-left: 0px\" width=800px>\n",
    "\n",
    "- **FAISS**: Meta 开源的向量检索引擎 https://github.com/facebookresearch/faiss\n",
    "- Pinecone: 商用向量数据库，只有云服务 https://www.pinecone.io/\n",
    "- **Milvus**: 开源向量数据库，同时有云服务 https://milvus.io/\n",
    "- Weaviate: 开源向量数据库，同时有云服务 https://weaviate.io/\n",
    "- **Qdrant**: 开源向量数据库，同时有云服务 https://qdrant.tech/\n",
    "- PGVector: Postgres 的开源向量检索引擎 https://github.com/pgvector/pgvector\n",
    "- RediSearch: Redis 的开源向量检索引擎 https://github.com/RediSearch/RediSearch\n",
    "- ElasticSearch 也支持向量检索 https://www.elastic.co/enterprise-search/vector-search"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb8bb6af",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "    <b>扩展阅读：https://guangzhengli.com/blog/zh/vector-database</b>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ddbe8b0",
   "metadata": {},
   "source": [
    "### 4.5. Chroma 向量数据库\n",
    "\n",
    "官方文档：https://docs.trychroma.com/docs/overview/introduction\n",
    "\n",
    "<img src=\"./assets/chroma.svg\" style=\"margin-left: 0px\" width=800px>\n",
    "\n",
    "**1. 什么是 Chroma？**\n",
    "\n",
    "Chroma 是一款开源的向量数据库，专为高效存储和检索高维向量数据设计。其核心能力在于语义相似性搜索，支持文本、图像等嵌入向量的快速匹配，广泛应用于大模型上下文增强（RAG）、推荐系统、多模态检索等场景。与传统数据库不同，Chroma 基于向量距离（如余弦相似度、欧氏距离）衡量数据关联性，而非关键词匹配。\n",
    "\n",
    "**2. 核心优势**\n",
    "\n",
    "- 轻量易用：以 Python/JS 包形式嵌入代码，无需独立部署，适合快速原型开发。\n",
    "- 灵活集成：支持自定义嵌入模型（如 OpenAI、HuggingFace），兼容 LangChain 等框架。\n",
    "- 高性能检索：采用 HNSW 算法优化索引，支持百万级向量毫秒级响应。\n",
    "- 多模式存储：内存模式用于开发调试，持久化模式支持生产环境数据落地。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab1e2eaf",
   "metadata": {},
   "source": [
    "### 4.6. Chroma 安装与基础配置"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10f5c7e6",
   "metadata": {},
   "source": [
    "**1. 安装**\n",
    "\n",
    "通过 Python 包管理器安装 ChromaDB："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "82b74daf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 安装 Chroma 向量数据库完成功能\n",
    "# !pip install chromadb"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba8a1313",
   "metadata": {},
   "source": [
    "**2. 初始化客户端**\n",
    "\n",
    "- 内存模式（一般不建议使用）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "66139b4f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import chromadb\n",
    "\n",
    "client = chromadb.Client()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbd64ad1",
   "metadata": {},
   "source": [
    "- 持久化模式："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "7c597ea4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 数据保存至本地目录\n",
    "client = chromadb.PersistentClient(path=\"./chroma\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6345fd2",
   "metadata": {},
   "source": [
    "### 4.7. Chroma 核心操作流程"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd388d07",
   "metadata": {},
   "source": [
    "**1. 集合（Collection）**\n",
    "\n",
    "集合是 Chroma 中管理数据的基本单元，类似关系数据库的表：\n",
    "\n",
    "（1）创建集合"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "e357d9d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from chromadb.utils import embedding_functions\n",
    "\n",
    "# 默认情况下，Chroma 使用 DefaultEmbeddingFunction，它是基于 Sentence Transformers 的 MiniLM-L6-v2 模型\n",
    "default_ef = embedding_functions.DefaultEmbeddingFunction()\n",
    "\n",
    "# 使用 OpenAI 的嵌入模型，默认使用 text-embedding-ada-002 模型\n",
    "# openai_ef = embedding_functions.OpenAIEmbeddingFunction(\n",
    "#     api_key=\"YOUR_API_KEY\",\n",
    "#     model_name=\"text-embedding-3-small\"\n",
    "# )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dea069a2-3c86-4925-b949-423225d97f26",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 自定义 Embedding Functions\n",
    "from chromadb import Documents, EmbeddingFunction, Embeddings\n",
    "\n",
    "class MyEmbeddingFunction(EmbeddingFunction):\n",
    "    def __call__(self, texts: Documents) -> Embeddings:\n",
    "        # embed the documents somehow\n",
    "        return embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "d66ee484-2de1-4002-8c1a-901730f0e3db",
   "metadata": {},
   "outputs": [],
   "source": [
    "collection = client.create_collection(\n",
    "    name = \"my_collection\",\n",
    "    configuration = {\n",
    "        # HNSW 索引算法，基于图的近似最近邻搜索算法（Approximate Nearest Neighbor，ANN）\n",
    "        \"hnsw\": {\n",
    "            \"space\": \"cosine\", # 指定余弦相似度计算\n",
    "            \"ef_search\": 100,\n",
    "            \"ef_construction\": 100,\n",
    "            \"max_neighbors\": 16,\n",
    "            \"num_threads\": 4\n",
    "        },\n",
    "        # 指定向量模型\n",
    "        \"embedding_function\": default_ef\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "914d7bd0",
   "metadata": {},
   "source": [
    "（2）查询集合"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "218e02ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "collection = client.get_collection(name=\"my_collection\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae319d34",
   "metadata": {},
   "source": [
    "- peek() - returns a list of the first 10 items in the collection.\n",
    "- count() - returns the number of items in the collection.\n",
    "- modify() - rename the collection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "151bac2c",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'ids': ['id1', 'id2'], 'embeddings': array([[ 1.60079654e-02,  1.02847122e-01,  1.15382075e-02,\n",
      "        -3.34708579e-02,  1.14530073e-02,  6.35105297e-02,\n",
      "         7.69377127e-02,  2.63175592e-02,  3.02551482e-02,\n",
      "         5.60058244e-02,  1.71475381e-01, -5.42212315e-02,\n",
      "        -1.58406403e-02, -6.51467144e-02, -1.57864839e-02,\n",
      "         2.83155753e-03, -3.26167569e-02,  2.66733784e-02,\n",
      "        -7.15856552e-02,  8.44806433e-03, -2.99695563e-02,\n",
      "        -2.17505582e-02,  5.32626212e-02,  3.43727171e-02,\n",
      "        -9.07171890e-02,  1.21168364e-02, -2.69576702e-02,\n",
      "         4.59876433e-02,  1.01122260e-01, -1.81238037e-02,\n",
      "        -5.19720837e-02,  2.03293059e-02,  4.75841053e-02,\n",
      "         3.78946075e-03, -8.18847271e-04,  1.22508323e-02,\n",
      "        -4.76963632e-02, -3.22107375e-02,  1.30394297e-02,\n",
      "         7.83042014e-02,  7.84149021e-03,  1.85555387e-02,\n",
      "        -3.15110246e-03, -7.80368447e-02,  4.13435362e-02,\n",
      "         4.90222089e-02, -4.06925753e-02,  3.93145084e-02,\n",
      "        -7.46777561e-03, -5.22653386e-02, -3.55002731e-02,\n",
      "         2.06290453e-04, -3.41081098e-02, -1.56410262e-02,\n",
      "         2.31633540e-02, -3.78748327e-02,  1.62366647e-02,\n",
      "        -6.90942407e-02, -4.53344663e-04,  3.18625532e-02,\n",
      "        -4.05511819e-02,  2.68111303e-02,  8.16123709e-02,\n",
      "        -8.30093399e-03, -2.61857919e-02, -5.55653274e-02,\n",
      "         8.82894918e-02,  3.47467773e-02, -3.39885689e-02,\n",
      "         6.67572841e-02, -3.69542986e-02, -6.84789121e-02,\n",
      "         1.05227130e-02, -5.18546291e-02, -2.79148985e-02,\n",
      "        -5.15082069e-02,  3.94463493e-03, -3.29275019e-02,\n",
      "        -4.12891805e-02, -4.97297049e-02, -9.03679151e-03,\n",
      "         1.55774383e-02, -1.54050114e-02, -2.44077388e-02,\n",
      "        -4.35537100e-02,  8.28523263e-02, -8.96914750e-02,\n",
      "        -6.56676441e-02,  9.06960592e-02, -1.72327831e-02,\n",
      "        -5.63939400e-02, -4.19318769e-03,  5.64119518e-02,\n",
      "        -2.60585137e-02, -5.36089651e-02,  7.63042131e-03,\n",
      "         5.73804490e-02, -2.93727499e-02, -5.99319898e-02,\n",
      "        -2.18895413e-02,  2.84335315e-02,  3.21940407e-02,\n",
      "         4.33418341e-02, -1.69674959e-02, -9.57008079e-02,\n",
      "        -9.64404419e-02, -5.58742359e-02, -4.14000303e-02,\n",
      "         7.42312009e-03,  2.13643312e-02, -1.27961570e-02,\n",
      "        -9.64226872e-02, -4.05461416e-02, -8.90704915e-02,\n",
      "         4.36905622e-02,  5.41711189e-02, -2.50783525e-02,\n",
      "        -6.20226599e-02, -4.31861132e-02,  8.00017640e-03,\n",
      "         4.43116352e-02,  2.16560817e-04, -6.78144768e-02,\n",
      "        -3.73440385e-02, -8.52543786e-02, -5.78837954e-02,\n",
      "         1.12444095e-01,  1.12699947e-34, -1.30182533e-02,\n",
      "         3.61953154e-02, -4.17085849e-02, -5.60728014e-02,\n",
      "        -3.53362896e-02, -7.25883022e-02,  2.83340625e-02,\n",
      "        -4.29319032e-03, -2.51931790e-02,  5.90070076e-02,\n",
      "         2.41631288e-02,  3.55208069e-02, -7.12787136e-02,\n",
      "         2.22141556e-02, -6.45648763e-02,  2.45486759e-02,\n",
      "        -2.25814953e-02, -2.90450919e-02,  6.93994015e-02,\n",
      "        -1.62690207e-02,  3.37140486e-02, -2.19794717e-02,\n",
      "         2.36658566e-02, -3.02177556e-02,  4.73420806e-02,\n",
      "        -6.37411028e-02, -8.61779135e-03, -3.19620408e-02,\n",
      "        -4.42778692e-02, -4.38023941e-04,  1.92251243e-02,\n",
      "         9.21577364e-02, -7.12817460e-02, -1.83417052e-02,\n",
      "        -4.57913913e-02, -5.55779822e-02, -4.75130379e-02,\n",
      "         5.78707829e-02, -2.57463418e-02,  9.16123688e-02,\n",
      "         1.44482665e-02, -6.86100572e-02, -1.51709309e-02,\n",
      "        -3.54547016e-02,  6.27886578e-02,  6.02783598e-02,\n",
      "         3.80713306e-02,  2.83817071e-02,  5.83127998e-02,\n",
      "        -7.50461780e-03, -4.38718088e-02, -4.42299135e-02,\n",
      "        -5.37308194e-02,  5.90511039e-02,  5.45813181e-02,\n",
      "        -4.43352573e-02, -7.24943951e-02,  6.33559283e-03,\n",
      "        -7.72805810e-02, -1.52551821e-02,  2.07857098e-02,\n",
      "        -7.64854029e-02, -6.60533458e-02,  4.69730720e-02,\n",
      "        -2.60253754e-02, -3.04362923e-02, -4.70122881e-02,\n",
      "        -3.10077965e-02,  6.94416985e-02, -8.46251622e-02,\n",
      "        -1.71880517e-02, -1.73908584e-02,  8.06723461e-02,\n",
      "        -1.98594248e-03,  1.60956327e-02, -2.91899964e-02,\n",
      "        -1.17881589e-01,  6.30488694e-02,  7.61399837e-03,\n",
      "         1.16791651e-02, -7.42094219e-02, -3.27940919e-02,\n",
      "        -1.46846110e-02,  4.90979142e-02, -2.80960668e-02,\n",
      "         7.04982206e-02,  1.02233805e-01, -3.22975069e-02,\n",
      "        -5.42287901e-02, -2.57338677e-02, -1.33626908e-01,\n",
      "        -2.05116253e-02,  7.59315491e-02,  6.01240993e-03,\n",
      "         1.41837951e-02, -5.65840699e-33,  5.25438040e-02,\n",
      "         2.52399202e-02, -7.55674988e-02,  2.67315619e-02,\n",
      "         7.34107196e-02, -1.85619444e-02, -6.07455336e-03,\n",
      "         1.01233616e-01, -5.02436459e-02,  2.60642283e-02,\n",
      "         1.64058730e-02, -6.06203005e-02, -1.95530728e-02,\n",
      "         4.11556885e-02,  1.65515300e-02,  4.51068319e-02,\n",
      "         1.92852672e-02,  6.75794259e-02, -5.66973574e-02,\n",
      "         3.02763786e-02,  2.08511185e-02, -6.19761832e-02,\n",
      "        -1.69548038e-02, -1.00626014e-02, -2.26544570e-02,\n",
      "         1.60595737e-02,  1.30055174e-01, -1.13482356e-01,\n",
      "         2.88127735e-02,  3.85586433e-02, -2.98077594e-02,\n",
      "         4.77387160e-02, -3.40080559e-02,  9.65462998e-02,\n",
      "         1.18568270e-02, -4.31572162e-02,  8.36032927e-02,\n",
      "        -5.45478500e-02, -1.32005453e-01,  1.73883308e-02,\n",
      "        -6.47106441e-03,  2.19145883e-02,  3.66144292e-02,\n",
      "        -2.35652942e-02, -2.10894216e-02,  9.75336879e-02,\n",
      "        -6.41137958e-02, -4.18875776e-02, -3.26700993e-02,\n",
      "        -7.52105331e-03,  1.18186712e-01, -5.46923764e-02,\n",
      "         1.55380890e-02, -7.09711388e-02, -8.93085543e-03,\n",
      "         6.07514083e-02, -5.48398448e-03, -3.97939794e-02,\n",
      "        -4.68779095e-02, -1.59829631e-02,  4.42453846e-03,\n",
      "         3.89415510e-02, -4.98985872e-02,  2.84185242e-02,\n",
      "        -3.24949622e-02, -3.14683393e-02,  1.83271598e-02,\n",
      "        -2.93961298e-02,  2.71999612e-02, -7.13919029e-02,\n",
      "         2.32536867e-02, -2.12188847e-02,  5.40255420e-02,\n",
      "        -9.17197764e-03, -1.20779730e-01, -1.38249239e-02,\n",
      "        -2.89596636e-02, -8.22960399e-03,  6.75575212e-02,\n",
      "         1.82665456e-02, -5.33265173e-02, -4.73627597e-02,\n",
      "         1.01903295e-02,  7.66777694e-02, -5.61003052e-02,\n",
      "        -6.63899407e-02, -4.71642241e-02,  3.38872224e-02,\n",
      "         2.63481569e-02,  3.35130990e-02,  9.88939404e-03,\n",
      "         9.11035854e-03, -1.46274529e-02,  1.31188622e-02,\n",
      "         4.51860912e-02, -2.95720692e-08,  2.42869575e-02,\n",
      "        -1.03651531e-01, -2.39919424e-02, -7.52649233e-02,\n",
      "        -6.04568515e-03, -4.67056669e-02, -5.93406782e-02,\n",
      "         2.31960807e-02,  1.52380383e-02,  5.41231111e-02,\n",
      "        -2.98468256e-03,  8.08415785e-02, -6.44019768e-02,\n",
      "         8.48919973e-02, -3.74744013e-02, -1.10662747e-02,\n",
      "        -5.75166605e-02,  7.09377378e-02,  2.22671796e-02,\n",
      "        -1.76407844e-01,  1.79807004e-02, -3.13793086e-02,\n",
      "        -8.14918336e-03, -1.49637476e-01, -4.58033942e-02,\n",
      "         4.82478552e-02, -1.14434905e-01,  2.66169175e-03,\n",
      "        -6.57881126e-02,  7.46465614e-03,  1.69582963e-02,\n",
      "        -5.18701551e-03,  3.46466005e-02, -6.63819835e-02,\n",
      "         7.12288963e-03,  4.14013257e-03,  1.16104618e-01,\n",
      "        -5.71295917e-02, -7.45662674e-03,  1.88890677e-02,\n",
      "         4.04121950e-02,  1.72094896e-03,  3.50083522e-02,\n",
      "         8.66099261e-03,  1.00683428e-01, -5.29526360e-02,\n",
      "         3.16514969e-02,  6.77122995e-02,  1.76563580e-02,\n",
      "         2.89173927e-02,  2.56963372e-02,  3.55979614e-02,\n",
      "         4.31170985e-02, -3.99968401e-02, -1.23511543e-02,\n",
      "         2.50801686e-02,  2.40242146e-02, -3.19561586e-02,\n",
      "         2.04282049e-02,  1.32023301e-02,  6.59063607e-02,\n",
      "         4.27544229e-02,  4.74615581e-02,  7.29628652e-03],\n",
      "       [-4.34405319e-02,  8.62214044e-02,  5.96775971e-02,\n",
      "        -1.26900766e-02,  2.51021050e-02,  4.05610055e-02,\n",
      "         1.33904815e-01, -8.47285613e-03,  2.17851903e-02,\n",
      "        -3.50075588e-02,  1.59151495e-01, -1.03970729e-01,\n",
      "         5.85770682e-02,  1.39493225e-02, -6.09262995e-02,\n",
      "        -1.72434337e-02, -2.28496362e-02, -4.87247929e-02,\n",
      "        -2.76443474e-02, -6.35287678e-03,  4.59020995e-02,\n",
      "         4.82951328e-02, -3.53034809e-02,  1.49584711e-02,\n",
      "         3.02953296e-03,  2.31359596e-03,  3.80055718e-02,\n",
      "        -7.12043121e-02,  1.19496800e-01, -6.62358850e-02,\n",
      "        -2.42726635e-02,  7.11036846e-02,  8.37838463e-03,\n",
      "        -4.66503995e-03,  6.66920766e-02,  2.34626979e-02,\n",
      "         2.07780115e-02, -7.22929835e-02,  4.23191227e-02,\n",
      "         5.30927107e-02, -7.63268117e-03, -5.07851988e-02,\n",
      "         1.53014343e-03, -5.90729900e-02, -1.78903546e-02,\n",
      "         4.41286899e-03, -1.80668067e-02,  4.43538874e-02,\n",
      "        -2.51900796e-02,  2.77939998e-03, -1.93474756e-03,\n",
      "        -4.46818359e-02, -4.22204547e-02, -7.68861845e-02,\n",
      "         2.49023307e-02, -1.35636339e-02, -5.23626693e-02,\n",
      "        -2.22924650e-02, -3.40574905e-02, -4.89343628e-02,\n",
      "        -5.06379306e-02,  5.14606573e-02,  3.60220820e-02,\n",
      "         3.96588743e-02, -7.63505744e-03,  5.13959564e-02,\n",
      "         2.26626005e-02,  1.36772795e-02,  9.64434352e-03,\n",
      "         2.83040479e-02, -3.71684842e-02,  4.55529895e-03,\n",
      "         2.66934428e-02,  1.05660381e-02, -6.94718063e-02,\n",
      "        -5.53349555e-02,  4.91293259e-02,  1.55150164e-02,\n",
      "        -1.12434858e-02, -1.08479008e-01, -3.44072692e-02,\n",
      "        -5.42641664e-03,  2.26589758e-02, -1.46892648e-02,\n",
      "         2.43695714e-02,  3.86866764e-03, -8.22533593e-02,\n",
      "        -5.04986430e-03, -2.74551641e-02,  6.30929321e-02,\n",
      "        -7.54046813e-03,  2.14150827e-02, -3.96481715e-02,\n",
      "         1.77627765e-02, -4.93343212e-02, -1.88094471e-02,\n",
      "         2.27969009e-02, -4.13041525e-02,  8.02531764e-02,\n",
      "         6.06909059e-02, -1.21676596e-02,  2.22857706e-02,\n",
      "        -6.42134901e-03,  4.30141017e-02, -8.03498551e-02,\n",
      "        -1.33757427e-01, -3.52484658e-02, -8.29417631e-02,\n",
      "         5.44838943e-02,  8.32328014e-03,  1.78216863e-02,\n",
      "        -1.32231578e-01, -6.04655966e-02, -1.15404502e-01,\n",
      "        -4.77960333e-02,  3.35868001e-02,  3.35066132e-02,\n",
      "         7.75430072e-03, -2.49594171e-02, -7.27711841e-02,\n",
      "        -6.46954682e-03, -3.35219130e-02, -8.86349007e-02,\n",
      "        -2.59677600e-02, -5.75712286e-02, -3.19023766e-02,\n",
      "         1.17316656e-01,  1.07744784e-33, -6.60240799e-02,\n",
      "        -5.87185938e-03, -5.79613373e-02, -2.36495603e-02,\n",
      "        -4.11779918e-02,  2.54827039e-03,  6.01726696e-02,\n",
      "        -1.14492858e-02, -9.22860280e-02, -1.51390927e-02,\n",
      "         1.71206072e-02,  1.00610264e-01, -3.51803117e-02,\n",
      "        -1.08585809e-03, -4.53209132e-02, -3.97195760e-03,\n",
      "         1.41077703e-02,  3.66324969e-02,  3.50605547e-02,\n",
      "        -4.57733199e-02,  4.21705209e-02, -8.96803364e-02,\n",
      "         1.40570896e-02, -2.78762691e-02, -1.77552290e-02,\n",
      "        -2.01198850e-02,  1.24351662e-02, -4.45588008e-02,\n",
      "        -2.16373727e-02,  2.72326730e-02,  5.51257841e-02,\n",
      "         6.00314466e-03,  2.51273699e-02,  1.26690790e-02,\n",
      "        -9.12073553e-02,  4.39566150e-02,  7.42829032e-03,\n",
      "         1.29825678e-02,  3.69849205e-02,  4.77660820e-02,\n",
      "        -4.03444543e-02, -5.95495179e-02, -2.59116245e-03,\n",
      "        -1.06615581e-01, -2.89349481e-02,  1.28350751e-02,\n",
      "         4.28387038e-02, -1.06545955e-01,  5.27544394e-02,\n",
      "        -4.62223291e-02,  8.94141197e-03,  2.13971292e-03,\n",
      "        -3.09102628e-02,  1.95975285e-02,  1.26316056e-01,\n",
      "        -1.28323548e-02, -1.22127011e-01,  4.61585596e-02,\n",
      "        -1.02815270e-01, -1.54136112e-02,  1.88882165e-02,\n",
      "        -5.31236678e-02, -7.89630637e-02,  2.78290436e-02,\n",
      "         4.39950600e-02,  1.67168584e-02, -1.21786855e-02,\n",
      "        -6.95747957e-02,  7.54998922e-02, -7.11238384e-02,\n",
      "        -6.64822087e-02, -3.46483327e-02,  6.66727275e-02,\n",
      "        -1.38775609e-03, -3.04548480e-02, -1.09602213e-02,\n",
      "        -5.78553602e-02,  6.55466691e-02,  5.62791936e-02,\n",
      "        -3.66934165e-02, -1.10830903e-01, -5.18629625e-02,\n",
      "         6.72017485e-02, -2.03972515e-02, -5.07389754e-03,\n",
      "         5.74801154e-02,  1.86637193e-02,  4.62966301e-02,\n",
      "        -1.55306617e-02,  1.05309905e-02, -9.62835625e-02,\n",
      "         1.06272334e-02,  2.20803786e-02,  4.90501784e-02,\n",
      "         1.14555480e-02, -6.34999183e-33, -2.62490604e-02,\n",
      "         8.05328488e-02, -1.01835288e-01, -3.99917327e-02,\n",
      "        -2.19847225e-02,  1.12269707e-02,  2.18730681e-02,\n",
      "         1.49087548e-01, -4.16750498e-02, -3.62580195e-02,\n",
      "        -7.93117285e-03, -1.88673027e-02, -4.20021601e-02,\n",
      "         3.83256376e-02, -8.27082023e-02,  4.58448799e-03,\n",
      "         5.50837368e-02,  6.31345436e-02, -4.31120768e-02,\n",
      "         3.33575755e-02, -5.00506861e-03, -8.04986209e-02,\n",
      "        -7.06320554e-02,  2.86188000e-03,  4.98594344e-02,\n",
      "         3.64037938e-02,  7.77288750e-02, -7.91657344e-02,\n",
      "        -1.38574960e-02,  3.64805050e-02, -3.17218751e-02,\n",
      "        -1.74664112e-03, -4.39159647e-02,  7.61267096e-02,\n",
      "        -2.26724278e-02, -1.26244217e-01,  7.59722665e-02,\n",
      "         1.71792097e-02, -5.25022373e-02,  7.81056806e-02,\n",
      "        -2.46431958e-03,  9.76559743e-02,  5.47443749e-03,\n",
      "        -4.83235344e-02,  8.77887756e-03,  7.15167820e-02,\n",
      "        -9.93312523e-03, -2.28629000e-02, -9.66208335e-03,\n",
      "        -1.85683733e-04,  5.82786091e-02, -1.61548778e-02,\n",
      "         7.38524124e-02, -3.40955593e-02, -5.83572239e-02,\n",
      "         5.40503189e-02, -4.86849062e-02, -6.51744530e-02,\n",
      "         1.69722270e-02,  5.37166558e-03,  3.80418152e-02,\n",
      "         5.82565516e-02, -4.79168259e-02,  4.02606614e-02,\n",
      "         1.65704228e-02, -1.99076976e-03, -8.14469624e-03,\n",
      "        -1.24649974e-02,  9.77152493e-03, -9.18249115e-02,\n",
      "         8.87049586e-02, -1.95588768e-02, -7.16136247e-02,\n",
      "        -1.24996761e-03, -9.22707617e-02,  6.14615791e-02,\n",
      "         1.57992914e-02,  1.67680737e-02,  3.29428203e-02,\n",
      "         2.96570938e-02,  5.37492037e-02,  5.31582348e-02,\n",
      "         3.07850577e-02, -3.37076001e-02, -4.30449983e-03,\n",
      "        -4.31761183e-02, -5.34105022e-03,  2.12001838e-02,\n",
      "        -7.41305947e-03, -5.00493590e-03, -4.72187158e-03,\n",
      "         5.07837720e-02, -2.57659405e-02,  5.58438571e-03,\n",
      "         4.46311757e-02, -3.20877227e-08, -9.47599411e-02,\n",
      "        -7.88708925e-02, -7.91676939e-02, -1.29080802e-01,\n",
      "         4.22643200e-02,  6.27878755e-02, -2.01248303e-02,\n",
      "        -3.39156985e-02,  1.57956928e-02, -9.72639471e-02,\n",
      "         1.69896577e-02, -9.20307040e-02, -7.89384916e-02,\n",
      "         2.62903478e-02,  1.82330851e-02,  4.27687104e-04,\n",
      "        -6.00656942e-02,  2.58287527e-02, -1.44665409e-02,\n",
      "        -7.24373758e-02,  1.12109957e-02, -2.03133617e-02,\n",
      "         1.11879967e-02, -6.16163872e-02, -1.56878754e-02,\n",
      "         3.56694087e-02, -8.38656630e-03,  1.91106983e-02,\n",
      "        -1.04408339e-03, -9.78449546e-03, -9.08549502e-02,\n",
      "         4.39148806e-02,  4.41064462e-02,  2.82191467e-02,\n",
      "         1.96879879e-02,  6.79198131e-02,  1.01080418e-01,\n",
      "        -2.10263897e-02,  4.63960646e-03,  1.07398257e-01,\n",
      "        -1.61623433e-02, -2.29236428e-02,  4.37740274e-02,\n",
      "        -2.04707794e-02,  1.16112210e-01, -5.14083467e-02,\n",
      "         6.07730597e-02, -1.92732178e-02, -1.58919301e-02,\n",
      "         4.92018973e-03, -1.39949908e-02, -2.78266408e-02,\n",
      "         1.05566178e-02, -2.98974402e-02, -2.70893704e-02,\n",
      "         3.89399305e-02, -9.87977441e-03,  2.00460758e-02,\n",
      "        -5.35070635e-02,  5.25764190e-02,  5.20156361e-02,\n",
      "         7.13807791e-02,  7.88660273e-02,  2.02222038e-02]]), 'documents': ['RAG是一种检索增强生成技术，在智能客服系统中大量使用', '向量数据库存储文档的嵌入表示'], 'uris': None, 'included': ['metadatas', 'documents', 'embeddings'], 'data': None, 'metadatas': [{'source': 'RAG'}, {'source': '向量数据库'}]}\n",
      "2\n"
     ]
    }
   ],
   "source": [
    "print(collection.peek())\n",
    "\n",
    "print(collection.count())\n",
    "\n",
    "# print(collection.modify(name=\"new_name\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c72168a",
   "metadata": {},
   "source": [
    "（3）删除集合"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "dbb92ce6",
   "metadata": {},
   "outputs": [],
   "source": [
    "client.delete_collection(name=\"my_collection\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b85787b6",
   "metadata": {},
   "source": [
    "**2. 添加数据**\n",
    "\n",
    "支持自动生成或手动指定嵌入向量："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "9c6a50d8",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Administrator\\.cache\\chroma\\onnx_models\\all-MiniLM-L6-v2\\onnx.tar.gz: 100%|█| 79.3M/79.3M [00:06<00:00, 13.8Mi\n"
     ]
    }
   ],
   "source": [
    "# 方式1：自动生成向量（使用集合指定的嵌入模型）\n",
    "collection.add(\n",
    "    # 文档的集合\n",
    "    documents = [\"RAG是一种检索增强生成技术\", \"向量数据库存储文档的嵌入表示\", \"在机器学习领域，智能体（Agent）通常指能够感知环境、做出决策并采取行动以实现特定目标的实体\"],\n",
    "    # 文档元数据信息\n",
    "    metadatas = [{\"source\": \"RAG\"}, {\"source\": \"向量数据库\"}, {\"source\": \"Agent\"}],\n",
    "    # id\n",
    "    ids = [\"id1\", \"id2\", \"id3\"]\n",
    ")\n",
    "\n",
    "# 方式2：手动传入预计算向量（实际开发中推荐使用）\n",
    "# collection.add(\n",
    "#     embeddings = get_embeddings(\"RAG是什么？\")\n",
    "#     documents = [\"文本1\", \"文本2\"],\n",
    "#     ids = [\"id3\", \"id4\"]\n",
    "# )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "086fd708",
   "metadata": {},
   "source": [
    "**3. 查询数据**\n",
    "\n",
    "- 文本查询（自动向量化）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "e0d0ac78",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'ids': [['id1']], 'embeddings': None, 'documents': [['RAG是一种检索增强生成技术，在智能客服系统中大量使用']], 'uris': None, 'included': ['metadatas', 'documents', 'distances'], 'data': None, 'metadatas': [[{'source': 'RAG'}]], 'distances': [[0.3491383194923401]]}\n"
     ]
    }
   ],
   "source": [
    "results = collection.query(\n",
    "    query_texts = [\"RAG是什么？\"],\n",
    "    n_results = 3,\n",
    "    where = {\"source\": \"RAG\"}, # 按元数据过滤\n",
    "    # where_document = {\"$contains\": \"检索增强生成\"} # 按文档内容过滤\n",
    ")\n",
    "\n",
    "print(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "348ad091",
   "metadata": {},
   "source": [
    "- 向量查询（自定义输入）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b5d79f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# results = collection.query(\n",
    "#     query_embeddings = [[0.5, 0.6, ...]],\n",
    "#     n_results = 3\n",
    "# )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91caef7b",
   "metadata": {},
   "source": [
    "**4. 数据管理**\n",
    "\n",
    "更新集合中的数据："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "83f97244",
   "metadata": {},
   "outputs": [],
   "source": [
    "collection.update(ids=[\"id1\"], documents=[\"RAG是一种检索增强生成技术，在智能客服系统中大量使用\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8b65e18",
   "metadata": {},
   "source": [
    "删除集合中的数据："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "c35bd06a",
   "metadata": {},
   "outputs": [],
   "source": [
    "collection.delete(ids=[\"id3\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d3c8991",
   "metadata": {},
   "source": [
    "### 4.8. Chroma Client-Server Mode\n",
    "\n",
    "- Server 端\n",
    "\n",
    "默认端口 8000\n",
    "```sh\n",
    "chroma run --path /db_path\n",
    "```\n",
    "\n",
    "修改端口：--port 端口号\n",
    "```sh\n",
    "chroma run --port 8001 --path /db_path\n",
    "```\n",
    "\n",
    "- Client 端"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8903a73d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import chromadb\n",
    "\n",
    "# 远程连接 Chroma Server\n",
    "chroma_client = chromadb.HttpClient(host='localhost', port=8000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b80fc459-7248-4a2d-950e-cef4d5dec2b3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n"
     ]
    }
   ],
   "source": [
    "# 测试\n",
    "collection = chroma_client.create_collection(name = \"my_collection\")\n",
    "collection = chroma_client.get_collection(name=\"my_collection\")\n",
    "print(collection.count())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e89911e8",
   "metadata": {},
   "source": [
    "## 5. Milvus 扩展学习\n",
    "\n",
    "中文官网：https://milvus.io/zh\n",
    "\n",
    "<p style=\"text-align:center\">Milvus 架构图<p>\n",
    "\n",
    "<img src=\"./assets/Milvus.png\" width=\"1200px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "027484f4",
   "metadata": {},
   "source": [
    "## 6. 学习打卡\n",
    "\n",
    "1. 掌握 Embedding 模型的原理及选型\n",
    "2. 安装 Chroma 向量数据库\n",
    "3. 使用 Chroma 本地持久化完成其核心操作\n",
    "4. 理解 Chroma Client-Server 模式\n",
    "5. 掌握 Mivus 向量数据库的使用（扩展学习）"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
